Repository navigation
[DNM][Experimental] Add B200 TP2 V4 Flash SGLang 8k1k MTP sweep / 新增 B200 TP2 V4 Flash SGLang 8k1k MTP 扫描 - #3517
functionstackx wants to merge 6 commits into
Conversation
新增 B200 TP2 DeepSeek V4 Flash serving 扫描,在共享可写目录下载模型并隔离 HF/Xet 缓存。
|
Thanks for the contribution!
中文感谢你的贡献!
|
将 B200 扫描变更记录关联至 PR #3517。
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=36328554889 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=36328554889 |
将 B200 Flash 镜像 digest 转为 Pyxis/Enroot 原生语法,保留固定 digest 和有效 squash 缓存优先级。
There was a problem hiding this comment.
I reviewed this PR and didn't find any bugs. Because it adds a new download-and-stage code path to a shared production launcher (runners/launch_b200-nscale-slurm.sh) that stages a large checkpoint onto shared infra storage before every SRT submission, a human look would still be worthwhile.
What was reviewed:
- The new
native-single-node/DeepSeek-V4-Flashbranch inlaunch_b200-nscale-slurm.sh: confirmed the per-commandHF_HOME/HF_HUB_CACHE/HF_XET_CACHEoverrides are scoped only to theuv tool run ... hf downloadinvocation and do not leak into the later containerHF_HUB_CACHEexport used for the shared mount. --dsv4passthrough insrt_fixed_sequence.sh: verified--dsv4is an existing, already-implemented flag ininfx/bench_serving/benchmark_serving.py, so this is a mechanical forward, not new client logic.- New
dsv4flashrecipe/config entries (configs/nvidia-master.yaml, the new8k1k.yaml): argument names (speculative-algorithm,cuda-graph-max-bs-decode, etc.) match patterns already used in sibling MTP recipes elsewhere in the repo. - Test coverage for the download success/failure paths in
test_srt_single_node.py, including exit codes, mount/model-path assertions, and cache isolation.
Extended reasoning...
The diff adds a new B200 fixed-sequence sweep config (YAML recipe + nvidia-master.yaml entry), forwards an existing --dsv4 client flag, and adds a checkpoint-download branch to the shared launch_b200-nscale-slurm.sh launcher with cache-path isolation, plus matching docs/changelog/tests. No auth, crypto, or injection-sensitive surface is touched; the main risk is operational (a new elif branch in a widely-shared launcher script and a download step onto shared storage) rather than security. New tests cover the download success/failure paths and appear consistent with the code, and the PR author's own description notes GPU memory/perf/accuracy validation is still pending, which is exactly the kind of production-launcher change worth a second pair of eyes even with no bugs found.
This review covers commit d24d369, which is no longer the latest commit on this pull request; later commits are not covered by it.
Set cuda-graph-backend-{decode,prefill}=disabled, which also skips target
verify and draft capture, and drop the per-concurrency decode graph batch.
|
Heads-up: #3576 (merged) replaced the bash launchers with a Python launcher, so this PR will conflict when you merge |
|
This pull request has had no activity (commits, comments, reviews or other updates) for more than 30 days, so it is being closed automatically. Reopen it if it is still needed. |
Description
Add
dsv4flash-fp4-b200-sglang-mtpfor the exactdeepseek-ai/DeepSeek-V4-Flashcheckpoint on B200 Nscale: TP2, 8192 input / 1024 output tokens, concurrency 1, 2, 4, 8, 16, 32, 64, 128.EAGLE, 2 steps, top-k 1, 3 draft tokens, not the separate EAGLE3 algorithm.cuda-graph-backend-decode: disabled,cuda-graph-backend-prefill: disabled), so decode, prefill, and EAGLE target-verify/draft all run eagerly; startup reports zero capture time for every graph phase.nightly-dev-cu13-20260925-8ca82118and amd64 digest. Keep weights and KV on GPU, real verification, and the checkpoint's shipped draft head. Explicitly disableSGLANG_NVFP4_CKPT_FP8_NEXTN_MOE; no engine patches, external draft, or draft-precision overrides./data/home/sa-shared/gharunners/models/DeepSeek-V4-Flash. Each launch resumes or validates the download. Failure stops submission.HF_HOME,HF_HUB_CACHE, andHF_XET_CACHEto writable paths below that checkpoint for the download process only. This avoids the inherited container-only/read-only cache paths that blocked the B300 download attempt.--dsv4client option throughsrt_fixed_sequence.shto use the DeepSeek-V4 chat encoder. This is the same shared client change as [DNM][Experimental] Add B300 TP2 V4 Flash SGLang 8k1k MTP sweep / 新增 B300 TP2 V4 Flash SGLang 8k1k MTP 扫描 #3516, included here so this PR can run independently.Upstream recipe reference: SGLang DeepSeek-V4 configuration. The sibling B300 recipe uses bundled EAGLE 3/1/4. This PR tests B200 TP2; memory fit, performance, and accuracy remain unverified until GPU CI passes.
Checkpoint metadata: DeepSeek-V4-Flash config. The draft uses the checkpoint's bundled head and pinned upstream default loading, without draft quantization or draft KV overrides.
Validation
cluster:b200-nscale, c1–128; c64/c128 selected for eval.RUNNER_NAME=local-validation, including real launcher execution with stubbed external tools, download-cache isolation, model-path forwarding, abort-on-download-failure, native digest conversion, and cached-image precedence.repo:tag@sha256:...as username credentials and received HTTP 400. The launcher now uses its existing Enroot conversion for this lane's uncached image, passingrepo:sha256:...while preserving the original image identity/digest and valid squash-cache precedence. Serving settings are unchanged. Replacement GPU validation is pending.git diff --checkpass.AI model disclosure
GPT 6 Astra Fast prepared the configuration, launcher/client changes, tests, PR, and CI monitoring. No delegated agents.
中文
新增
dsv4flash-fp4-b200-sglang-mtp,在 B200 Nscale 上使用原始deepseek-ai/DeepSeek-V4-Flashcheckpoint,运行 TP2、8192 输入 / 1024 输出 Token,并发为 1、2、4、8、16、32、64、128。用户已确认沿用 B300 PR #3516 的原生 MTP 设置:
EAGLE、2 steps、top-k 1、3 draft tokens,不是独立 EAGLE3 算法。沿用官方 SGLang CUDA 13 nightlynightly-dev-cu13-20260925-8ca82118及 amd64 digest,权重和 KV 保留在 GPU,采用真实验证,保留 checkpoint 自带 draft head。显式关闭SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE,不修改引擎、不使用外部 draft、不覆盖 draft 精度。cuda-graph-backend-decode: disabled、cuda-graph-backend-prefill: disabled),decode、prefill 及 EAGLE 验证/draft 均以 eager 模式运行;启动日志中各 graph 阶段捕获耗时均为 0。SRT 提交前将未预置的 checkpoint 下载到
/data/home/sa-shared/gharunners/models/DeepSeek-V4-Flash,每次启动均续传或验证下载,失败时停止提交。仅下载进程将HF_HOME、HF_HUB_CACHE和HF_XET_CACHE指向该 checkpoint 下的可写路径,避免继承容器专用或只读缓存路径。服务进程的容器缓存设置不变。共享固定序列客户端透传已有
--dsv4参数,使用 DeepSeek-V4 chat 编码器。此修改与 #3516 相同,在本 PR 中保留以便独立运行。扫描仅覆盖新增 B200 配置,默认在 c64/c128 执行 eval,不请求合并。验证:矩阵生成得到八个 TP2 8k1k 点;设置
RUNNER_NAME=local-validation后 112 项针对性 SRT 测试通过,涵盖下载缓存隔离、模型路径传递、下载失败时停止提交、原生 digest 转换和缓存镜像优先级;Ruff、Bash 语法和 diff 检查通过。B200 TP2 的显存容量、性能与准确率仍待 GPU CI 验证。首次
full-sweep-enabled扫描已完成 checkpoint 下载,但 Pyxis 将 Docker 的repo:tag@sha256:...解析为用户名凭据,导致 HTTP 400。现已在该路径调用启动器已有的 Enroot 转换逻辑,将未缓存镜像传为repo:sha256:...,保留原始镜像身份、digest 和有效 squash 缓存优先级。Serving 设置不变,修正后的 GPU 验证仍待完成。AI 模型使用说明:GPT 6 Astra Fast 负责配置、启动器与客户端修改、测试、PR 和 CI 监控,未委派其他 agent。